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to be small in distance by utilizing a reference 
image obtained in the past, or 

by making a search in a limited search area for 
searched points on an epipolar 
line . 

SOLUTION: A brightness-change detecting part 21 
computes a brightness- 
change evaluation value SD 
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imm i ] mmmmzx o jt«3*i*:B«tc 

uE^ii^iwi . mim^mz x o w Entitle l s 

t nr&B 3 ] mimsMm^mz * o ** 

nm^mt. mzwmmzx <ommmm$ 

zmizxfrtem&nmizmzi-fz z t tmt-t 
&mm i tzimmimmmm. 

4*. frieL#^<l<J:0^# : 5:ffltL-C^$^m2 
<r> I » v J: 0 i /hS v WW* 4* 2 <^)WBf# 

t sitat-t6i»*«2t:iEa«oBtt«saeii. 

21 L T J5r5£*>BflMQB £fttB««i^*8B«!!i WT 

U£»fl8!*4»&H«3ftfcB«tf>'3 ^^=5r< 1 1> 1 o 
U 8ftJM3ftT^4*i|iB*i:ji*fc 

lienwaai*^ "y7Tji:a{ sn^ttt: J: o . xe#- 40 

t £*tr£ fc *Hai:-$-4B«8!SEi£. 

nix m fcnmwttm tnt m&>m&mz . 

Wie»flaBl*»ftJltRJf*i3tB«^ a ti>li 

zmm&ti. rnmmztLT^&mmmtm&z 

77^ >-±^WJi5^cofll^iIS:^g-ri,^XT 7 7* 50 
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*7P^?A*IWtt4£fc$1«*fc*4«tt«W. 

[000 1] 

«MI£fflvvC. IBlwSflaF/hS n*>sa»£fflSrU 
*«lSft. Sfla&WS ofcflKSftfcBSSfctfLTU:. 

3 l=T4 d b izX *) . EEftBttO&^aaigafl: 

s^nE(c-r4BflasisaitjJ:t/*a. ^tafias* 

KRW4. 

[0002] 

*** T-oraewii* 4*4 iBtttiMgfftffti &*vt «- > 
[0003] 01 ATW&vmzviffltim 

>51t#.^^72^2^^^7l;J: , 5, 3^7C<7) 

is-y**mi. MMixd bt mmco3<X7tmmz 

[0004] .015 ^^-tMTIi . 3 ^7C>- 

^Cnbt. «RWr^<52fc:J: l 3«!lS*l4i»Ejanrfc*« 

[0005] BS^inb IZtt&i-hWM&nt *«{B-f4 

■«r*>t». E 1 5 i o 5 2co^^ 

n r (i. W*^5«03lfi^*'ll» (3t»> tS?ffi*^5lcO 

B«fflj^i?*B«±fc#fc*-* . ^^ttH^'. xtJK 

-77^WlfilJ. S«^7UM^72« 

{iaw^7>'«^t'S)^«f . 5 i m®m*mz 

mitl* 7 2 OBffi±<?)X 7 7 W yJiT'^^JE^i 
■CS 4. 

[0006] 016 ( a ) tc^u/;»ira± 
<0£nb(7)*fJ&ii£. <WBBBJiT«iaj'f**&feK^ 
•tS. 016 (A)<?)J:9K. nbMUa«0/|^«{WS-f 
V7V- bbl. «ajB«±Ox 9 y±os 
It* (01 6 (B)c0^r&. ndi^nd606A) T-ffl 

M n d 6 0^^£7)|iC^<i . *#l-f*l«lt* ^ 5 1 *» OH 
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3 

i. 

[0007] znttmrnwuk lx\z, mu£&& 

IZTFfX oliZSAD (Sum of Abusolute Difference ) tfi 
d (b, d) =S II (xi) - I' ( x' i ) | 

zvx&tz&vc * i ( xi ) imwm&<mmi&. r 

<x' i)tt#S®M«»K£*U itt>MK«Wrt*> 
B*»#t£*|-rfrc*>4. 
[00 08] ±*tlO*«)feiifcfllWfi*^57K:L 

EH7T'J>&„ -TW^KiiVvt. fttfflU 10 

Kii. iH3a*£4-f*3mi:&5. ifctt. gtfflWeo 
1 705%g\ no.) . 

[ 0 0 0 9 ] x£#-7 7>f >±«Oi4St8j*<7) 

tBHMWfi&JtLZtli. 20 
[00 10] 

xu, 1^-77^ y±^^f±6^-C'fc-?^^\ 

a# 3 2 AM 6 4 ffl«oaSRAt J: 0 . ttJ&&liS*3fi 
5. Sf /»Ki<0»*tt. S*BflLhO#B*. £Tfc*t 

[ 0 0 1 1 } 4fc. BSHWrirTU. BK*WW-*fc 
*>. Bffc£«l£-f &B3&0&#£0. B«s£# 30 

[0012] *«BHUC«J: o**afc«»T*5nfc 

£ftfc!0IB*0>B*S:Jll^4. 4fcti. xtjJf-55 

& z k iz i o . mum^^mm^smim h x o 

[00 13] 40 

[ ism &£*>?)#&] N$gi (ctestoBttffl 

9£gE(i* S^Hfr&««3ftfcB«^fc*J' 1 *<i: 

*nt hzt . 

[0014] Wii<iR5(cieewB«^s^ii, a«g 

k U $&ilft3tiTViftroBfl^&£t31ttS*ifc 50 
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mil: mmogHtin&tmm&AT r 

77^ y±0*tJtE^^iEBSri^-tl.!*SXT-yr 

[0015] mm^zwmmmmi.. aa&M& 

WflEWifi* r ••/ a S fifcffifc J: 0 . x tr* 1 - 7 5 
[0016] !f#JS 1 £IEtt*)B{ftWIgS. 5 

3*1, Soffit i 0 . Xh^-77-f V±C0MJtD 

[00 17] 

[ «BB<^**t«jKB] vmzxmnmcommw 

*>8M>JSSl*lC* ttJ&ti5tJlW>JB» (fiL-W) *tt 

[0018] mem i H^jra^ati, mmm^m \ 

^^*^Tv^aeiiB«fca**cwii$<ifc»PB« 

t«ai*«S5fl:*jicaji-*«s»aj#a mm. mi 

tt3&fft*£#B 01 3«9Xx'y7S34, 

S35) t*fili6ifct»afci-S. 
[0019] IWJ3I2 CEne)Btt®I£iEti:. 

tv7S3) fca&icfli*, itog^aii, wiir^atj; 

(«^.tf> 04WXr-xrS4) iktwat-r*. ! 
[0020] IS^JS 3 cettaB(M®Ii&Stt , WBdl i 

*>/h*v^53&»*¥iIBf"fSflBf^« (Witf, 01 3co 

^.r7TS3 3) sr^fetcfigi., jftse^au. mm 

(«itf. 01 30XT-/7-S3 4) ClfcSrlfmtt 
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C002 1 ] w*iB4 cettaiftffia&iKii. mm 

£ufl (#Jx.(4*. 014^SDt2) XVii'hZ^fr&fr 
S 5 5 ) ZZUzffit. *te£##(4. »2J0WBf#Hfc 

(Witf. 014co*T-y7 , S5 6) 

t-rs. io 

[ 0 0 2 2 ] 0 Hi. *%UiaBtt&I£K2 O^SfiX 
01£jSl/O>&. »#.*7 lTSHftSfUfc«¥ttl 5 

COlHlT-^ii. 7U-A;**y 6tC. 7 2? 

6. 7i'-A**y5tcEtt3ftfcBtiftT-*li. £ft 
B«co£*fcfflv>6#i4. 

[0023] 7P-io<*';6fcKtt3*i.fcB«r-* 20 
(4. BE«M«±**3fc, 7W-A.X*y7fcHtfj;&*i 
6. *LT. ^B«£tfaB3fcttJflS*ifcB*?-* 
(4. flw>T-*£4:*>fc. f££tSIESfB«<o£j£fcffl 
v^ft. 7i^-A**y 7£ft;ft£ft£B&T-?(4. 

[0 0 24] 7P-A**y 7fcEtt§flfcB*7*-* 
(4. BSiB«4«*3ttll^$*i&. lgMBB±j£«3 

(4. A^ft^B^T-^gt. &MWm±& 
L. ftTTf*. iit, 80§tB«fc(4. #B*OflWfc& { 

OftM 7 1 ) friaCOBKfc*fJW-.&«lC*r> 30 
TV^6Bft***-fS. Sf0iB«4(S:a53(C{i. 7)y- 

'J8CX*S*i. EttSfiSr-^i. SE«B<i±l£SS 
3Kfewc. l7^-AttKSB:&3fLfcH3IBfR'C*> 
6. 

[00 2 5] ?V-J±*"EV7, 8(4. -eft-fit 2 tfc<9 
^y^«£ft£4:?(::LTt>l|V\ -eLT. -eft 

-t 6 i 1 1= i f? . t y *«ff -s t v > s jaatiMffc-* 

*.=5rnj:9fc-*4.ri:. &#«{ctt. «BLT^6B* 40 

[00 2 6] 112(4. 01£SLfcB«*BB§!iB2Ot 

2?J ( t ) . -f-eo 1 7P-A(fj£0B#fJ ( t - 1 ) . *<n 1 
y\y-Mk<r>m\ < t + l ) co. *fl*$JC&ftftBaffi 
«6Bt2 0<OT-^<OS(tK**LTV^. 
[ 0 0 2 7 ] I**) ( t - 1 ) CiJtt* . flflm£l2 
0«S$*^7 l*»fccoB«T-*tt. r^;Wb§ 
ft. 7U-A>*y6fc:JWj3*l. EltSftS. **3E 
tt&hfcr-** I B ( t - 1 ) b-fl Btt, 50 
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(4. *<Q7-9tffte>tLfcmm7&tt>0>b-i-&) . R 
«(:, #5H*^9 2*»fe^B«f f -^i. ri-^MtS 
ft. SCAJiSft. EttSfi*. e-cofa 

ttsfifcT*-** id ( t - i ) tr* (Jaai d(4. 

«H*^9 2TflM63iifcB« J f--^*SL. MMfi 

(4. e<9T-?co#^ft£ii»^%<ot-f.i>) . 
[0028] 7i/-A^ty 7t(4. mm(t-i) co 

1 7P-Ajtll<7)M«T-? (8#M ( t - 2 ) ?. 
^51A»feffi*3*lfcB«T-^) ?*4IB(t- 

2) mmzti-za*). *coe*s*vo*&t-*jWE 
KB«±jsa3tai*sn£. 7k-^t ystii. 

WfM ( t - 1 ) COl 7U-AUC0E«B« ( t- 

2) r. imwi%&j8.m3tfftiiitzmMW®) x-hi 

R (t-2) j&sfHSSJiTtiO (£M*R(i. gggiB^ 
jS»33Wt*Lfcffi)liB«*^L. fSMl*l(4. e-coBBSt 
Bfl^f»6*ifc«*i!l*^fc<9fc-t4) . -tcoEttSft 

[0029] mmw®£j&&3\i. x-hztitt-tz 

at. SEStB&R ( t - 1 ) fc£s£U ttflrf*. dco 

[00 30] (t ) fcfcWC. 7V-A**y 5t 
(4. 4«fiWr^9 23i>6<OB«T-^ I D ( t ) 
ft. E«§*i*. 7U-A**y6£»i. 

1A^coB«t-^ I B ( t ) *«A*S<t. ie« 
3*1*. H^^lz-^t'J 5, 6(CIB1i$ix^ 
B«7*-^ ID ( t ) , I B ( t ) 14. raiBflt^K 
3KAfl3#U». 

[003 1 ] 7U-A^ i &y7tti. BWI(t-l)<0 
l#K7U~A^=£y 6*»^a}*?<tfcB«f-^ I B 

(t-1) EttS*t, *LT. KtBflte 

**3tifJ**<l4. 7I^-A^^y8t(4. «*](t 
- 1 ) co^traiB«4«»3*»4>aj*S*ifcKKBiR 
R ( t - 1 ) *«7Jj3*u IBISSn. ^LT . SEffiM^ 

4je«3fcaj*siis. 

[0032] JS£WM®£M®3X'te. XJlZixtz&T- 
^^S(=. ^fi ( t ) C£f?*£IIEBftR ( t ) SHJ^j 

[0033] (t + l) (IfcV^T. 7V-A^tl) 
5 (C(4. #t!S*^ 7 2 *>£>c9BffiT-* ID ( t + 1 ) 

tfAiizti. imztiz. mmz. 7v-A^^y6(; 

(4. »*^7 l*»fctf>B«T-? IB ( t + 1 ) 
UZil. EfiSft.*. *il-WWD7l'-A^tU5. 6 
KEH^tHfeBftT*-^ I D ( t + 1 ) , I B ( t + 
1 ) (4. ffi«Bfl&J**3t=AflS*l4. 
[00 34] 71^-A^^y 7(C(4. B#M ( t. ) C0B#(C 
71^-A^^ y 6*»^UJ**itfcB«'5 f -^ I B ( t ) 
EWSfi. 8gMBffi^«g|53(cai 
7]$flS. 71/- A>^ 'J 8tli, ^IJ ( t ) C0B*CE 
KB«4lS*3*»<baj^S*in:EKB«R ( t ) *«A* 



10/23/2003, EAST Version: 1.4.1 



7 

&. 

[0035] KE»B«£lS*3Ttt. AflSilfc&T- 

* Mt. ttJBl ( t + 1 ) (cfettftEKBttR ( t + 
1 ) fcttWrT*. 

[00 36] ClWiotl/t. A^'J 5, 6t 
14. *^MCJ:4B«T---*#12tt3fU *0M9»J: 

'J 7 ffiSIWfll**7 U- A** 'J 8fc. **V? 

[0037] EBtBft±j£S3li, H3 t^LfcJ: a 

XTVitmm2 3frm$Li*tvc^&. vxr. mm 
( t ) nmzmizmx. swfti. mm ( t ) tt$v^ 
t , ns&£ftmn&2 1 t:a. aep*> y i *»&<ob« 

IB ( t ) t , l$M ( t - 1 ) £*5tt43PB 

[0038] »SS»b«kiti»2 1 li. A^Sil^S-T- 

«ttfl<OSM»HittSDitiW|[tttl*"t 
4. iOfMittSDitWM ( t- 1 ) tfc^tSEMH 
&£j£&3 (-*fM-*9I«2 3) A^aiaSflfcEK 
H&R ( t - 1 ) *\ £§R8ie&£tf 2 2 CCA^Sft 
4. »««BI»ffi«2 2li, A*§*ifc?-*&afc. 

<O^Sftfc»SMHfflfc , I D ( t ) Kfto-C, XfU 
*«UI»2 3Ji. BEKBftR ( t ) 
[0 0 39] 04(07 0-^-^- hSr#ffBLT. 
EKB«4««3<0«^fc:oi,>-CK"»t4. I 
Bi. iDi. Rilifft-fh, &i#g<9B£fcijlt 

3^;&.x?lfr4>0Bft*MML #8SfoM5 2j&>4> 
«B*«IWE. BftSGftOfi (Hat) £«U **>fttc 

[00 40] EEStB$l4ifiJca3ti. ^f77Sll;fcu 

t . Bsw>*9*«-r i * o Kwatss-r-f. . nmmt 

C XtlZtlfzl B (t) c7)HiSeoii*WT-^ I 
Bi ( t ) t . IB ( t — 1 ) OH i #a<0B*Or- 

* I Bi ( t - 1 ) fceoHSrjggU SS£Bfli<Ofl£<D 
35fl:«SDi**ftS. 

SDi = | I Bi (t ) - I Bi ( t-1 ) I 
[004 1] C<0»gc^flHiSDic7)Wai^tt*"i: L 

*> 0 £ . **i?ft«B^H2l^J^*>«IMtt K4 

[0 0 4 2] ®mm®M%22lZ. Xr-yrS3lCfc 

mmcitmm s d i l £ v mi s d t i »h v > t 

DitfLt'VMiSDti^hS^tt'pCifcfis Uligg 
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«B«T<oe«^«(Wb (**>B*«>«(r*!i ( t - 1 ) izti 
nzmmtrnm ( t ) tzm&mM<r)&) umx-zi 
iiir/h* wc** o tvmzti. mm(t-i) nmiz 

1tfcftfc»i#B<3B*fc»ittf4IE«i«Ri (t- 
l ) £ , Bft! ( t ) agi SBaBXCtt&f &EBB 

«Ri (t ) tLTSfflTI.. 
[0 04 3 J ^f7rS3MW. tt&gftff 

-3\ it«ia> *i#a<oB«T^aaBS'(W4**v^ 

10 TTb^fcPJKafifc*^ ^•r-y7'S5t:atr. ^-r 

••/7-S5tCfcV^T» XTU-^I^2 3(i % #!d^^7 
2c0ffi^r-^ I D ( t ) c7)Xh-7K-5 5^f y±<9*f)fc 

[0044] 06 K^-f i o 3i$Sc£ISfcgAtfi*: 
J lHHft»l»^5H:»jSWr^52W26f5^9 (* 
20 fV^^5) -CBKUfc*^, *5SB 

\t, m$cmizx>ommz$em&x'hi. 029, 3 
ft (JMt\ aenB«fcas«Ei-*) TnbCiais 

a, #JH^>9 20B« (WT. «HB*i*fflHE* 
4) •CnrKBKSflfc*^, nt*>feniA^aiS3 

x 3<Dm$cmrm h t . ifcet^iti. . 

n t =H ■ nt ( 1 ) 

[ 0 0 4 5 ] * ( 1 ) *>»|0E»frflHli. mourns 
30 ^'4>, ^^-/l«?Ki6**<«4<?)T, S**{i8 

[0 04 6] @7{i, St^^2:Jfi-r»^fc : 5r4f-^ 

Hffi*^ Siifc2^0*^5t:J: l JJt«S/i4J:dt 

40 T :3 cX J ?- J rtfJ8f$.ZtlX 

I 0 0 4 7 ] 5 20M^t . 7 1 COB 

A (tfflll) It. 08 (A) K*-f<HC&4. 
[0048]I^t^, *MHB*fc*tLTBfJg*>WI85 
«ISri6L. i^fe^Bflraw^^BUbWR/jNfcOrfiJ: 

?i] H $r$a6 4 1 1 iz Jt 0 . SOT«Off*»jftK» Jt?f 

50 [0049] ^smnmsamt . mmm&tf-ftti 
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»»3E»fT*IH m9&*tfk&*«aJi&) kl 

Xi±^ Levenberg-Marquardtft/jN-fbftAifflV^^I). •£ 

[00 50] 09(2. xW-77^f y<7)S^$riiBH 

mm Z o KfcJt 43HWflLJ:<OiS n b t=«lSt &#Siii« 
±Oj&nrortn&££ftl>. jS[nbU:ffirc*.S0)T\ * 

com. mmm^^zco^znmmmmm 

fc& J: 3 Kii*>*ifc«^eM«0afttfM Hi tffol . 

[oo5i] mmm»±<nwmtin b ii. 

Ho, HifciO. #SaHflLh«Onro, n t i C-fil-efltt 

[ o o 5 2 ] k z bx\ 9&tm9LW>wk*ns?tt 

jS) #SIB«<0xt!K-55-f fi-CfiSRU 20 

n t 2?:^<J6. Zil^nia, n r i, n I 2* > 

(AC/BC) / (AD/BD) 
= (sina/sin/3 



* [oo53]iaio{i, wb&msthi&tffflvh 

t. zcomiz^tkolz. 3ftji&ffifoco&#Limi& 

<Ofc#, fil^L±«04*A. B.C. Dti. *il-F*u 
B^L" ±0>j6A' , B' , C ' , D' tft»S*lT^ 

W&Ok. mL, L' fcttH-¥liiJitSBS*iTv> 

-6. jwrn^-st. £a. b. c. d. a* . b' . 

C ' , D ' kM&Okli. R-WTiLhfcEBStiTV^ 
1 He^-TJ; 3 

[0054] £0)IHCi>l,vC. OA = a, OB = b, O 
C = c, OD = dkti%. ZhlZ. AOC = a, BOC 
= 13, AOD = r . BOD = 5"i:fc<i: s ^il^W^W 
HffiB. OTO^tc J: fc tfX'Z h . 

AC/BC=AOAC/AOBC (2) 
AOAC= ( 1/2 ) acsina (3) 
AOBC= ( 1/2) bcsin/3 (4) 
[ 0 0 5 5 ] it ( 2 ) JM ( 4 ) (I J 0- OTOBWfcj; 

AC/BC = asina/bsin/S (5) 
[00 5 6] |5l««ft^ADi;BDtWUTSrri:. 

AD/BD = asinr/bsin<5' (6) 

[oo57]5U5) *s*(6) izxmm&zkiz 

J: 1 ). (7) ##£>il&. 



/ (sinr/ sind 1 ) 



(7) 



Itt*. A' C , B' C 



w (7) <3tt*fcB«0> 30^L-Cff3^i:(cJ;0. OTO* ( 8 ) tf&t>ti&. 
A' D" . B' D" izttx 



(A' C /B' C ) / (A' D' /B' D' ) 

= (sina/sin/3 ) / (sinr/sini? ) 
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[00 59] Ztlh. 5U7) i::* (8) fcfrfc, OTA 
(AC • BD/BC • AD ) 
= ( A' C ' ■ B 
[0060] JJLbOftJtfcfflVvC, OT<OJ: SSr^Jfifc 
J: 0 » rm<MtML&0) 3 aScSEHrt tt4 
*3l0tj!i«*tf>t*iS. v^, HI lfcfc^T, *8«-L* 

A, B, DZZtlZmmZo. Z 2 , Zi093&7c£[I 
flOj£t*fjS$-tf\ £A' , B" . D" IfflJ 

£*ffMHo. Hz, HiT^ft^';h.:£V>S£fc#T'#& A 
AC = Zs-Zo , A' C 
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* NOTICES * 

Japan Patent Office is not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[The technical field to which invention belongs] this invention relates to an image processing system, a 
method, and an offer medium, especially The criteria picture acquired in the past in the search of 
corresponding points called for in case a depth map is generated is used. As opposed to the pixel which 
judged whether change of distance would be small, consequently was judged that change is small The 
search range of the corresponding points is related with the image processing system which enables 
improvement in the speed of generation processing of a depth map, a method, and an offer medium 
limitation or by being made not to search at all. 
[0002] 

[Description of the Prior Art] In order to measure, surface 3-dimensional coordinate, i.e., 3-dimensional 
configuration, of a measuring object object, it asks for the corresponding points of each pixel between 
the picture with two or more cameras with which views differ using the picture which picturized the 
same scene simultaneously, and the distance metering device which acquires the distance information to 
a body with the parallax is known. Thus, generally the method of measuring distance is called stereo 
method. 

[0003] Drawing 15 is drawing explaining the outline of the stereo method. As shown in this drawing, by 
the stereo method, with two cameras, the criteria camera 1 and the reference camera 2, a 3-dimensional 
scene is observed and the position (distance from a reference point) in the 3-dimensional space of the 
object which it is going to measure is called for. 

[0004] That is, in the example shown in drawing 15 , the viewpoint nb by which the object point P in a 
3-dimensional scene is observed with the criteria camera 1, and the viewpoint nr observed with the 
reference camera 2 are called for. And it can ask for the position in the 3-dimensional space of the object 
point P from these stations nb and nr. 

[0005] There is a method using the epipolar line as a method of detecting the viewpoint nr 
corresponding to Viewpoint nb. That is, as shown in drawing 15 , the viewpoint nr of the reference 
camera 2 exists on the straight line which the flat surface determined by the optical center (optical axis) 
of both cameras and the viewpoint nb of the criteria camera 1 and the picture side of the reference 
camera 2 cross. This straight line is called epipolar line. If the physical relationship of the criteria 
camera 1 and the reference camera 2 is ****, desired corresponding points are detectable by performing 
corresponding-points reference on the epipolar line on the screen of the reference camera 2 for every 
station of the criteria camera 1 . 

[0006] Here, the case where the corresponding points of the point nb on the criteria screen shown in 
drawing 16 (A) are detected on a reference screen is explained. Like drawing 16 (A), the small field W 
of nb circumference is made into a template, and a correlation value is calculated by two or more [ on 
the epipolar line on a detection picture ] (in the case of drawing 16 (B) ndl or six points of nd6). The 
number of the point ndl on this epipolar line or the subscript of nd6 presupposes that it is attached 
corresponding to lm of actual distance from the criteria camera 1, and 6m, respectively. 
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[0007] As how to calculate this correlation value, as shown, for example in the following formula, SAD 
(Sum of Abusolute Difference) is used. 

d(b, d) =sigma |I(xi)- I'(x'i) | -- in this formula, I (xi) expresses the brightness of a reference picture and 
the brightness value of a criteria picture and its I' (x'i) is [ i ] the values showing the number of the pixel 
in the small field W 

[0008] Drawing 17 made the graph the correlation value calculated by the upper formula. In this graph, 
a mutually related low and nd3 which it is at the time serve as corresponding points most. In this case, 
distance is set to 3m in which nd3 exists. Or most, the sampled data are interpolated, a minimum point is 
calculated from the value of the low nb3 mutually related circumference, and it is good also considering 
the point as corresponding points (in the case of drawing 17 ndm). 

[0009] In addition, the relation between the position on an epipolar line and an objective distance is 
beforehand called for by the calibration. And a depth map is generated by doing in this way and finding 
the corresponding points of the pixel on each criteria picture, and its distance. 
[0010] 

[Problem(s) to be Solved by the Invention] By the way, although there were six searching points on an 
epipolar line, 32 or 64 searching points usually search for corresponding points in the example of 
drawing 16 , Furthermore, this search is performed to each pixels of all on a criteria picture. Therefore, 
since this corresponding-points search was very heavy processing, there was a technical problem that 
time was required in the processing. 

[001 1] Moreover, at a broadcasting station, in order to think quality of image as important, there are 
many pixels which constitute a picture. Therefore, about the whole picture, time is required for 
performing corresponding-points search mentioned above and calculation of distance, and processing of 
real time becomes difficult. 

[0012] this invention enables it to accelerate generation processing of a depth map by it being made in 
view of such a situation, using the criteria picture acquired in the past, using the pixel of the depth map 
called for in the past to the pixel judged that change of distance is small, or limiting the search range of 
the searching point on an epipolar line, and making it search. 
[0013] 

[Means for Solving the Problem] An image processing system according to claim 1 makes a criteria 
picture at least one of the pictures picturized from image pck-up equipment, and is characterized by to 
have a determination means determine the search range of the corresponding points on an epipolar line 
with the value computed by a brightness calculation means compute change of the brightness of the 
criteria picture picturized now and the criteria picture picturized in the past, and the brightness 
calculation means. 

[0014] The image-processing method according to claim 5 makes a criteria picture at least one of the 
pictures picturized from image pck-up equipment, and is characterized by to include the determination 
step which determines the search range of the corresponding points on an epipolar line with the value 
computed at the brightness calculation step which computes change of the brightness of the criteria 
picture picturized now and the criteria picture picturized in the past, and the brightness calculation step. 
[0015] The brightness calculation step which computes change of the brightness of the criteria picture 
which an offer medium according to claim 6 makes a criteria picture at least one of the pictures 
picturized from image pck-up equipment, and is picturized now, and the criteria picture picturized in the 
past, It is characterized by offering the program which the computer which performs processing which 
contains the determination step which determines the search range of the corresponding points on an 
epipolar line with the value computed at the brightness calculation step can read. 
[0016] In an image processing system according to claim 1, the image-processing method according to 
claim 5, and an offer medium according to claim 6, at least one of the pictures picturized from image 
pck-up equipment is made into a criteria picture, change of the brightness of the criteria picture 
picturized now and the criteria picture picturized in the past is computed, and the search range of the 
corresponding points on an epipolar line is determined by the computed value. 
[0017] 
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[Embodiments of the Invention] Although the gestalt of operation of this invention is explained below, it 
is as follows, when the gestalt (however, an example) of operation [ in the parenthesis after each means ] 
is added and the feature of this invention is described, in order to clarify correspondence relation 
between each means of invention given in a claim, and the gestalt of the following operations. However, 
of course, this publication does not mean limiting to what indicated each means. 
[0018] Claim 1 image processing system makes a criteria picture at least one of the pictures picturized 
from image pck-up equipment. With the value computed by a brightness calculation means (for 
example, step S32 of drawing 13 ) to compute change of the brightness of the criteria picture picturized 
now and the criteria picture picturized in the past, and the brightness calculation means It is 
characterized by having a determination means (for example, steps S34 and S3 5 of drawing 13 ) to 
determine the search range of the corresponding points on an epipolar line. 

[0019] It has further a judgment means (for example, step S3 of drawing 4 ) to judge whether an image 
processing system according to claim 2 has the value smaller than the set-up threshold computed by the 
brightness calculation means, and a determination means is characterized by the thing using the distance 
information drawn by the corresponding points called for in the past (for example, step S4 of drawing 
4 ), when a value is judged to be smaller than a threshold by the judgment means. 
[0020] It has further a judgment means (for example, step S33 of drawing 13 ) to judge whether an 
image processing system according to claim 3 has the value smaller than the set-up threshold computed 
by the brightness calculation means, and a determination means is characterized by what (for example, 
step S34 of drawing 13 ) the search range of corresponding points is limited for around the 
corresponding points for which it asked in the past, when a value is judged to be smaller than a threshold 
by the judgment means. 

[0021] The value computed by the brightness calculation means an image processing system according 
to claim 4 The 2nd threshold set up as a bigger value than a threshold (for example, SDtl of drawing 
14 ) It has further the 2nd judgment means (for example, step S55 of drawing 14 ) which judges whether 
it is smaller than (SDt2 [ for example, ] of drawing 14 ). a determination means When it is judged by the 
2nd judgment means that a value is smaller than the 2nd threshold, it is characterized by what (for 
example, step S56 of drawing 14 ) the search range of corresponding points is limited for around the 
corresponding points for which it asked in the past. 

[0022] Drawing 1 shows the example of composition of the image processing system 20 of this 
invention. The image data of the photographic subject 15 by which a photographic subject's 15 image 
data picturized with the criteria camera 1 was picturized by the frame memory 6 with the reference 
camera 2 is digitized by the frame memory 5, respectively, and is outputted and memorized. The image 
data memorized by the frame memory 5 is outputted to the depth map generation section 3. The image 
data outputted to the depth map generation section 3 is used for the generation of a depth map later 
mentioned with other data. 

[0023] The image data memorized by the frame memory 6 is outputted to the depth map generation 
section 3 and a frame memory 7. And the image data which the image data outputted to the depth map 
generation section 3 was used for the generation of a depth map later mentioned with other data, and 
was outputted to the frame memory 7 is memorized. 

[0024] The image data memorized by the frame memory 7 is outputted to the depth map generation 
section 3. The depth map generation section 3 generates and outputs a depth map based on each inputted 
image data. Here, a depth map means the picture from which the brightness of each pixel is a value 
corresponding to the distance from a reference point (criteria camera 1). The output from a frame 
memory 8 is also inputted into the depth map generation section 3. It is inputted into this frame memory 
8, and the data memorized are the depth map outputted one frame ago in the depth map generation 
section 3. 

[0025] Frame memories 7 and 8 may be made to consist of two memory, respectively. And it can avoid 
rewriting making it not affect the processing which each memory is performing, and the data of the pixel 
specifically referred to by being made to perform writing and read-out in each memory by turns. 
[0026] Drawing 2 is drawing for explaining the image processing system 20 shown in drawing 1 along 
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with a time-axis, and shows the data flow of the image processing system 20 in each time of the time (t) 
used as criteria, the time in front of one of them (t-1), and the time after one of them (t+1). 
[0027] The image data from the criteria camera 1 of an image processing system 20 in time (t-1) is 
digitized, is inputted into a frame memory 6 and memorized. The memorized data is set to IB (t-1) (IB 
shall show henceforth the image data picturized with the criteria camera 1, and the inside of a 
parenthesis shall show the time when the data was obtained). Similarly, the image data from the 
reference camera 2 is digitized, is inputted into a frame memory 5 and memorized. The memorized data 
is set to ID (t-1) (ID shall show henceforth the image data picturized with the reference camera 2, and 
the inside of a parenthesis shall show the time when the data was obtained). 

[0028] IB (t-2) which is the image data (image data outputted from the criteria camera 1 at time (t-2)) 
one frame before time (t-1) is memorized by the frame memory 7, and the data memorized is outputted 
to the depth map generation section 3. R (t-2) which is a depth map (depth map which the depth map 
generation section 3 outputted at time (t-2)) one frame before time (t-1) is memorized by the frame 
memory 8 (R henceforth), the depth map which the depth map generation section 3 generated shall be 
shown, and the inside of a parenthesis shall show the time when the depth map was obtained ~ the depth 
map memorized is outputted to the depth map generation section 3 

[0029] The depth map generation section 3 generates and outputs a depth map R (t-1) based on the 
inputted data. About this generation, it mentions later. 

[0030] In time (t), image data ID (t) from the reference camera 2 is inputted into a frame memory 5, and 
it memorizes. Similarly the image data IB from the criteria camera 1 (t) is inputted into a frame memory 
6, and it memorizes. Image data ID (t) memorized by each frame memory 5 and 6 and IB (t) are inputted 
into the depth map generation section 3. 

[003 1] The image data IB (t-1) outputted from the frame memory 6 at the time of time (t-1) is inputted 
into a frame memory 7, and it memorizes, and is outputted to the depth map generation section 3. The 
depth map R (t-1) outputted from the depth map generation section 3 at the time of time (t-1) is inputted 
into a frame memory 8, and it memorizes, and is outputted to the depth map generation section 3. 
[0032] In the depth map generation section 3, depth map R (t) in time (t) is outputted based on each 
inputted data. 

[0033] In time (t+1), image data ID (t+1) from the reference camera 2 is inputted into a frame memory 
5, and it memorizes. Similarly the image data IB (t+1) from the criteria camera 1 is inputted into a frame 
memory 6, and it memorizes. The image data ID (t+1) and IB (t+1) memorized by each frame memory 5 
and 6 is inputted into the depth map generation section 3. 

[0034] The image data IB (t) outputted from the frame memory 6 at the time of time (t) is inputted into a 
frame memory 7, and it memorizes, and is outputted to the depth map generation section 3. Depth map 
R (t) outputted from the depth map generation section 3 at the time of time (t) is inputted into a frame 
memory 8, and it memorizes, and is outputted to the depth map generation section 3, 
[0035] In the depth map generation section 3, the depth map R (t+1) in time (t+1) is outputted based on 
each inputted data. 

[0036] Thus, the image data based on the time is memorized by frame memories 5 and 6, the image data 
obtained at time one frame ago rather than the time is memorized by the frame memory 7, and a depth 
map is memorized by the frame memory 8, respectively. 

[0037] The depth map generation section 3 consists of a brightness change detecting element 21, the 
search range determination section 22, and the stereo processing section 23, as shown in drawing 3 . 
Hereafter, the time of time (t) is mentioned as an example, and is explained. In time (t), the image data 
IB from the criteria camera 1 (t) and the image data IB (t-1) from the criteria camera 1 in time (t-1) are 
inputted into the brightness change detecting element 21 . 

[0038] The brightness change detecting element 21 calculates and outputs the brightness value-change 
evaluation value SDi based on each inputted data. The depth map R (t-1) outputted from the depth map 
generation section 3 (stereo processing section 23) in this evaluation value SDi and time (t-1) is inputted 
into the search range determination section 22. The search range determination section 22 determines 
the search range of the corresponding points on an epipolar line based on the inputted data. The stereo 
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processing section 23 asks for depth map R (t) according to the determined search range and ED (t). 
[0039] Next, with reference to the flow chart of drawing 4 , operation of the depth map generation 
section 3 is explained. EBi, EDi, and Ri shall express the brightness of the picture from the criteria 
camera 1 in the i-th pixel, the brightness of the picture from the reference camera 2, and the value 
(distance) of a depth map here, respectively, and the inside of the parenthesis added after that shall 
express time. 

[0040] The depth map generation section 3 carries out initial setting of the i showing the number of a 
pixel to 0 in Step SI. In Step S2, as shown in the following formula, the brightness change detecting 
element 21 calculates the difference of the inputted data EBi (t) of the i-th pixel of IB (t), and the data EBi 
(t-1) of the i-th pixel of IB (t-1), and calculates the change value SDi of the brightness of a criteria 
picture. 

SDi =|IBi(t)-IBi(t-l) | [0041] You may make it take the correlation value of the surrounding small field 
of each pixel as the method of calculation of the change value SDi of this brightness to the change which 
takes the absolute value of the difference of the brightness value of each pixel. 
[0042] It judges whether the reference range determination section 22 has the brightness change 
evaluation value SDi smaller than the threshold SDt set up beforehand inputted in Step S3. When it is 
judged that the brightness change evaluation value SDi is smaller than a threshold SDt, it progresses to 
step S4. Saying that the brightness change evaluation value SDi is smaller than a threshold SDt It is 
judged that change (difference of the distance in the time (t-1) of the pixel and the distance in time (t)) of 
the distance in the i-th pixel will probably be so small that it can be disregarded. The depth map Ri (t-1) 
corresponding to the i-th pixel obtained at the time of time (t-1) is adopted as a depth map Ri (t) 
corresponding to the i-th pixel of time (t). 

[0043] When the brightness change evaluation value SDi was judged to be larger than a threshold SDt, it 
put in another way in Step S3 and it is judged on the other hand that the distance change by the i-th pixel 
will probably be large, it progresses to Step S5. In Step S5, the stereo processing section 23 performs 
corresponding-points search on the epipolar line of image data ID (t) of the reference camera 2, and 
processing of calculation of distance. Next, although the detail of processing of Step S5 is explained 
with reference to the flow chart of drawing 5 , the method of calculation of the distance at the time of 
using drawing 6 or subsequent ones and using projective transformation before it, is explained. 
[0044] As shown in drawing 6 , when the flat surface put on 3-dimensional space is observed with two 
cameras (stereoscopic camera), the criteria camera 1 and the reference camera 2, the camera coordinate 
of corresponding points can be mutually changed by projective transformation. That is, the following 
formula will be materialized if the projective-transformation matrix of 3x3 is set to H for the conversion 
to nr from nb when the point P on the flat surface put on 3-dimensional space is observed by nb by the 
picture (it is hereafter written as a criteria picture suitably) of the criteria camera 1 and is observed by nr 
by the picture (it is hereafter written as a reference picture suitably) of the reference camera 2. 
nr-H-nb(l) 

[0045] The projective-transformation matrix H of a formula (1) is constituted by a total of nine 
parameters of mO or m8. However, since flexibility remains in a scale factor, flexibility is set to 8. 
Moreover, the projective-transformation matrices H are the internal parameter of a camera, an external 
parameter, and a matrix that contained the plane equation implicitly. 

[0046] Drawing 7 is drawing showing the relation of the texture and the camera used as the object which 

performs projective transformation. In this drawing, it is made as [ picturize / with two cameras with 

which physical relationship was mutually fixed in the flat surface of one sheet installed in the position 

which only a certain distance Z0 separated from the reference point (you may choose it as a position) ]. 

In addition, predetermined texture is formed in the field by the side of a plane camera. 

[0047] A stereo pair is formed combining the picture of the reference camera 2, and the picture of the 

criteria camera 1 . At this time, the picture (criteria picture) of the criteria camera 1 and the picture 

(reference picture) of the reference camera 2 come to be shown in drawing 8 (A). 

[0048] At this time, predetermined projective transformation is performed to a reference picture, and the 

projective-transformation matrix H in which the error of the brightness between these pictures serves as 
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the minimum is computed. By searching for such a projective-transformation matrix H, it becomes 
possible to search the corresponding points corresponding to the arbitrary points of a criteria picture 
from a reference picture. 

[0049] The Levenberg-Marquardt minimizing method is used as how (the method of a picture doubling 
lump) to search for the projective-transformation matrix H whose reference picture and criteria picture 
after conversion correspond. And an epipolar line is called for using the projective-transformation 
matrix H searched for in this way. 

[0050] Drawing 9 is drawing for explaining how asking for an epipolar line. If the projective- 
transformation matrix H is searched for, the point nrO on the reference picture corresponding to the point 
nb on the criteria picture in distance Z0 will be determined. Since Point nb is arbitrary consequently, the 
corresponding points corresponding to all the points on a criteria picture are computable. The projective- 
transformation matrix HI at the time of similarly, setting the flat surface to observe so that it may 
become the flat surface and parallel which were put on the position of distance Zl in the position of 
distance Z0 is searched for. 

[0051] When the viewpoint nb on a criteria picture has an object point in distance Z0 or Zl, the 
projective-transformation matrices HO and HI searched for show being projected on nrO and nrl on a 
reference picture, respectively. Therefore, the segment on a reference picture which connected two nrO 
and nrl serves as an epipolar line. 

[0052] By the way, when measuring the distance of an actual object, it is necessary to search for the 
point (corresponding points) corresponding to the predetermined point on a criteria picture on the 
epipolar line of a reference picture, and to change the searched position (parallax) of corresponding 
points into an actual distance in 3 -dimensional space. However, when projective transformation is used, 
for example, parallax and an actual distance are not in direct proportion. Therefore, in order to ask for 
the transformation which changes this parallax into distance, the above-mentioned flat surface is again 
installed in distance Z2 which is different in distance Z0 and Zl, it is asked for nr2, and it changes into it 
from these [ nrO nrl, and nr2 ] to the distance of parallax using a double ratio. 
[0053] Drawing 10 is drawing for explaining a double ratio, it is shown in this drawing - as -- the 
segment in 3 -dimensional space — L shall be projected as segment L' on a picture screen this time -- a 
segment -- 4on L A, and B, C and D -- respectively -- Segment L -- it shall be projected on 'upper point 
A, B', C, D' Moreover, View O, and Segment L and L' which observe these segments L and L ! are 
arranged at the coplanar. putting in another way - if -- a point ~ A ~ B ~ C ~ D - A -- ' ~ B ~ ' - C ~ ' 
„ D -- ' -- a view ~ O -- being the same - a flat surface -- a top -- arranging -- having ..****. When the 
physical relationship on these flat surfaces is rewritten, it comes to be shown in drawing 1 1 . 
[0054] In this drawing, it sets with OA=a, alumnus=b, OC=c, and OD=d, and further, if it sets with 
AOC=alpha, BOC=beta, AOD=gamma, and BOD=delta, the following formulas can express the relation 
between these. 
AC/BC=**OAC/**OBC (2) 
**OAC=(l/2) acsinalpha (3) 
**OBO(l/2) bcsinbeta(4) 

[0055] The following relational expression is obtained by a formula (2) or (4). 
AC/BC=asinalpha/bsinbeta (5) 

[0056] The following formulas will be obtained if same calculation is given to AD and BD. 
AD/BD=asingamma^sindelta (6) 

[0057] The following formulas (7) are obtained by carrying out the division of the formula (5) by the 
formula (6). 

(AC/BC) /(AD/BD) =(sinalpha/sinbeta)/(singamma/sindelta) (7) 

[0058] more than -- a formula -- (-- two --) -- or -- (-- seven --) - calculation -- being the same -- 

calculation ~ AC -- ' -- B'C -- ' -- AD -- ' - B'D ~ ' - receiving -- carrying out -- things - the following 

~ a formula - (-- eight --) -- obtaining ~ having . 

(AC/B'CytAD'/B'D') 

= (sinalpha/sinbeta) /(singamma/sindelta) (8) 
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[0059] The following formulas (9) and (double ratios) are obtained from these formulas (7) and a 

formula (8). 

(AC-BD/BC-AD) 

- (A'C'andB'D'/B'C'andA'D') (9) 

[0060] The above double ratio is used and the distance from the reference point in the 3 -dimensional 
space of a desired object point is found by the following procedures, now and drawing 1 1 -- setting -- a 
segment — L is corresponded to the visual axis of the point nb on a criteria picture — making — 
moreover, a segment — it carries out to making L' correspond to an epipolar line furthermore — a point -- 
A — B — D -- respectively -- distance -- Z — zero -- Z -- two -- Z — one ~ three — a dimension space -- 
inside ~ a point ~ corresponding - making -- moreover ~ a point - A -- ' -- B — * - D - * - projective 
transformation » a matrix ~ H — zero - H - two - H - one - respectively -- it can ask ~ reference ~ a 
picture — a top — a point -- nr -- zero -- nr -- Drawing 12 replotted these relations anew. 
[0061] Suppose that the object point C is arranged now in the position which only distance ZS left, 
supposing the corresponding points which searched the epipolar line L' top and were obtained are nrs at 
this time -- a formula (9) -- each the value of a segment is as follows 
[0062] 

AOZS-Z0 , A'C'=nrs-nr0 BC=ZS-Z2 , B'C'=nrs-nr2 AD=Z1-Z0 , A'D'=nrl-nr0 BD-Z1-Z2 , B'D'=nrl- 
nr2 (10 

[0063] Therefore, the distance ZS from the reference point to the object point C can be found by 
substituting a formula (10) for a formula (9). by the way, an actual distance — a segment although it is 
not a thing in alignment with L, since it projects orthogonally in another segment and the ratio of each 
distance serves as a relation of direct proportion, a big problem is not produced For example, in drawing 
12 , AD is the length of ACDC projected orthogonally in the optical axis. 

[0064] Moreover, although the case of interpolation between AD(s) was explained in the above example 
(when the object point C exists inside AD), it cannot be overemphasized that it is possible to apply the 
above principle similarly in the case of extrapolation (when for the object point C to exist in the outside 
of AD). 

[0065] Below, processing of Step S5 of drawing 4 is made into the object point corresponding to the i-th 
pixel for the object point C of drawing 12 , and it explains with reference to the flow chart of drawing 5 
by making into an example the case where the corresponding points and distance are computed. 
[0066] Execution of this processing inputs the object point nb (the object point C on a picture) on a 
criteria picture into the stereo processing section 23 in Step S21. In Step S22, the corresponding points 
nrO on a reference picture or nr2 is computed using the projective-transformation matrix HO searched for 
as mentioned above, or H2. And it progresses to Step S23. 

[0067] In Step S23, an epipolar line is generated by connecting nrO and nrl which were calculated in 
Step S22 in a straight line. And in Step S24, the pixel group near the point nb of the picture outputted 
from the criteria camera 1 ( drawing 1 ) is extracted. 

[0068] In Step S25, the extracted pixel group is made into a template and the point mrs that the degree 
of match is high is searched along with an epipolar line. That is, in Step S23, the processing in Step S25 
reads the pixel group currently outputted from the reference camera 2 according to the address 
corresponding to the generated epipolar line, and turns into processing collated with a template. And the 
center of a field that the nearest (there are few errors) pixel group exists is made into Point nrs. 
[0069] In Step S26, the distance ZS corresponding to nrs is computed using nrs, nrO, nr2 and Z0, or Z2 
to the double ratio called for in Step S25. The distance ZS (depth map Ri corresponding to the i-th pixel 
(t)) to the object point C (object point corresponding to the i-th pixel) is computable with such 
processing. 

[0070] there is no processing of this flow chart a lameness crack in Step S3 to the pixel by which the 
evaluation value SDi of brightness was judged to be smaller than a threshold SDt (therefore, search is 
not performed to all pixels) -- it is ~ it becomes possible to shorten the time which generation of a depth 
map takes 

[0071] The depth map Ri corresponding to the i-th pixel called for as mentioned above (t) is written in 
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depth map R (t) in Step S6 of drawing 4 . 

[0072] In Step S7, it is judged whether it was searched for i showing the number of a pixel to all the 
pixels that constitute a screen. If it puts in another way, it will be judged whether in the case of 640x480, 
the total of the pixel which constitutes a screen became [ the value of i ] equal to 640x480. In Step S7, 
when it is judged that the value of i is not equal to the total of a pixel, it progresses to Step S8 and the 
increment of the value of i is carried out only for 1. And it returns to Step S2 and processing after it is 
repeated. 

[0073] On the other hand, when it is judged in Step S7 that the value of i became equal to total of a 
pixel, processing of this flow chart is ended. 

[0074] Next, with reference to the flow chart of drawing 13 , other operation of the depth map 
generation section 3 is explained. Since processing of Steps S31 and S32 is the same processing as 
processing of Steps SI and S2 of drawing 4 , the explanation is omitted. 
[0075] It judges whether the reference range determination section 22 has the brightness change 
evaluation value SDi smaller than the threshold SDt set up beforehand inputted in Step S33. When it is 
judged that the brightness change evaluation value SDi is smaller than a threshold SDt, it progresses to 
Step S34. Saying that the brightness change evaluation value SDi is smaller than a threshold SDt It is 
judged that change (difference of the distance in the time (t-1) of the pixel and the distance in time (t)) of 
the distance in the i-th pixel will probably be small. The search range of the corresponding points on an 
epipolar line is set up around it focusing on the depth map Ri (t-1) corresponding to the i-th pixel 
obtained at time (t-1). 

[0076] When the brightness change evaluation value SDi was judged to be larger than a threshold SDt, it 
put in another way in Step S33 and it is judged on the other hand that the distance change by the i-th 
pixel will probably be large, it progresses to Step S3 5. In Step S3 5, the search range of the 
corresponding points on an epipolar line is set as the whole measurement range. 
[0077] In Step S34 or Step S3 5, the result of the search range on the set-up epipolar line is outputted to 
the stereo processing section 23 from the search range determination section 22. The stereo processing 
section 23 performs corresponding-points search on the epipolar line of image data ID (t) of the 
reference camera 2, and processing of calculation of distance in Step S36 based on the inputted search 
range. Henceforth, since Step S36 or processing of S39 is Step S5 of drawing 4 , or the same processing 
as processing of S8, the explanation is omitted. 

[0078] With the gestalt of operation mentioned above, although the search range of corresponding points 
was searched for only using one threshold SDt, the case where it carries out using two thresholds SDtl 
and SDt2 is explained with reference to the flowchart of drawing 14 . 

[0079] Since processing of Steps S51 and S52 is Step SI of drawing 4 , or the same processing as 
processing of S2, the explanation is omitted. It judges whether the search range determination section 22 
has the brightness change evaluation value SDi smaller than a threshold SDtl drawn at Step S52 in Step 
S53. 

[0080] If it puts in another way when it is judged that the brightness change evaluation value SDi is 
smaller than a threshold SDtl, change of the distance in the pixel will progress to Step S54, when it is 
judged that it is within the limits which can be disregarded. In Step S54, it adopts as a depth map Ri (t) 
corresponding to the pixel of the No. i side which outputs the depth map Ri (t-1) corresponding to the i- 
th pixel outputted from the depth map generation section 3 in time (t-1) from the depth map generation 
section 3 in Time t, and progresses to Step S59. 
[0081] In Step S59, the adopted depth map Ri (t) is written in R (t). 

[0082] On the other hand, in Step S53, the brightness change evaluation value SDi is equal to a 
threshold SDtl, or when it is judged that it is larger than it, it progresses to Step S55. In Step S55, it is 
judged whether the brightness change evaluation value SDi is smaller than a threshold SDt2. Let this 
threshold SDt2 be a larger value than a threshold SDtl. 

[0083] When it was judged that the brightness change evaluation value SDi is smaller than a threshold 
SDt2, are put in another way and it is judged that change of the distance in the pixel will probably be 
small, it progresses to Step S56. In Step S56, the search range of the corresponding points on an epipolar 
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line is set up only around it focusing on the depth map Ri (t-1) corresponding to the i-th pixel for which 
it asked at the time of time (t-1). 

[0084] On the other hand, in Step S55, the brightness change evaluation value SDi is equal to a 
threshold SDt2, or when it is judged that it is larger than it, it progresses to Step S57. In Step S57, the 
search range on an epipolar line is set as the whole measurement range. 

[0085] In Step S58, processing which computes corresponding points and distance is performed 
according to the search range set up in Step S56 or Step S57. Hereafter, since processing of Step S58 or 
Step S61 is the same processing as processing of Step S5 of drawing 4 , or Step S8, the explanation is 
omitted. 

[0086] Thus, in case stereo processing is performed and it asks for a depth map, using the picture and 
distance information which were searched for in the past, by limiting the search range on an epipolar 
line, the computational complexity of corresponding-points search can be reduced and improvement in 
the speed of processing can be attained. 

[0087] In the gestalt of operation mentioned above, although two cameras (the criteria camera 1 and 
reference camera 2) were used, even if it uses two or more two or more cameras, it cannot be 
overemphasized that this invention is applicable. Moreover, although the one depth map generation 
section 3 may be used as mentioned above, you may make it use two or more depth map generation 
sections 3. When using two or more depth map generation sections 3, each depth map generation section 
3 is constituted so that a part of picture assigned, respectively may be taken charge of and processed and 
it can process to parallel further. Thus, by being constituted, it can ask for a depth map early further. 
[0088] In addition, the transmission medium by networks, such as the Internet besides information 
record media, such as a magnetic disk and CD-ROM, and a digital satellite, is also contained in the offer 
medium which provides a user with the computer program which performs the above-mentioned 
processing in this specification. 
[0089] 

[Effect of the Invention] According to an image processing system according to claim 1, the image- 
processing method according to claim 5, and the offer medium according to claim 6, like the above At 
least one of the pictures picturized from image pck-up equipment is made into a criteria picture, and 
change of the brightness of the criteria picture picturized now and the criteria picture picturized in the 
past is computed, with the computed value Since the search range of the corresponding points on an 
epipolar line was determined, it becomes possible to become possible to shorten the time which 
corresponding-points search takes, to have and to accelerate generation processing of a depth map. 



[Translation done.] 
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